<html>
<head><meta charset="utf-8"><title>Pipelining for Buck · t-compiler/wg-pipelining · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/index.html">t-compiler/wg-pipelining</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html">Pipelining for Buck</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="166114738"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166114738" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166114738">(May 20 2019 at 19:55)</a>:</h4>
<p>The pipelining work is very exciting - I've been thinking/talking about it, but hadn't realized that there had been so much substantial work. I'm interested in extending Buck to support it. What should I be looking at? What caveats are there (aside from "all very experimental/unstable, expect it to stabilize in some other form")? Is there anything that's deeply, intrinsically Cargo-specific?</p>



<a name="166123964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166123964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166123964">(May 20 2019 at 22:10)</a>:</h4>
<p><span class="user-mention" data-user-id="213049">@Jeremy Fitzhardinge</span> the design currently matches pretty closely with the <a href="https://github.com/rust-lang/compiler-team/blob/master/working-groups/pipelining/NOTES.md" target="_blank" title="https://github.com/rust-lang/compiler-team/blob/master/working-groups/pipelining/NOTES.md">initial notes</a>, nothing is really Cargo specific in the sense that it's just flags in rustc really</p>



<a name="166123973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166123973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166123973">(May 20 2019 at 22:10)</a>:</h4>
<p>Buck may or may not architecturally be ready for the model of pipelined compilation we're using though</p>



<a name="166123978"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166123978" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166123978">(May 20 2019 at 22:10)</a>:</h4>
<p>which is that halfway through a step's execution it can produce an artifact which is consumed by further steps</p>



<a name="166123999"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166123999" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166123999">(May 20 2019 at 22:11)</a>:</h4>
<p>so you basically have to watch a process, wait for a signal via stdout that a file is ready, go spawn more work, and then continue to wait for the original process</p>



<a name="166124010"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166124010" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166124010">(May 20 2019 at 22:11)</a>:</h4>
<p>and then also make sure you synchronize correctly when linking and all that (wait entirely for dependencies rather than just their metadata)</p>



<a name="166124639"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166124639" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166124639">(May 20 2019 at 22:21)</a>:</h4>
<p>I think I can model it as two separate build actions which internally depend on a single rustc invocation, so each can "finish" when the corresponding artifact is ready.</p>



<a name="166124741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166124741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166124741">(May 20 2019 at 22:23)</a>:</h4>
<p>One design change I'd like to see: the <code>artifact</code> message should also have which thing is being emitted rather than having it being implied by the filename extension. eg, <code>{"artifact":"/Users/jsgf/git/cargo/target/release/deps/libsemver-46a2b7dea0b8356b.rmeta", "emit": "metadata"}</code>(to match the <code>--emit</code> option).</p>



<a name="166124864"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166124864" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166124864">(May 20 2019 at 22:25)</a>:</h4>
<p>The "two action" model will naturally get the deps right, so long as I make a library build depend on the rmeta, and the link depend on the rlib (and the partial ordering of all the rmeta -&gt; rlib dependencies will make everything else work out).</p>



<a name="166126039"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166126039" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166126039">(May 20 2019 at 22:43)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span>  ^</p>



<a name="166127348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127348">(May 20 2019 at 23:05)</a>:</h4>
<p><span class="user-mention" data-user-id="213049">@Jeremy Fitzhardinge</span> ah yeah this is the original model I implemented in Carog</p>



<a name="166127400"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127400" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127400">(May 20 2019 at 23:06)</a>:</h4>
<p>we ended up not landing it because it was too invasive and had weird repercussions elsewhere in Cargo</p>



<a name="166127405"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127405" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127405">(May 20 2019 at 23:06)</a>:</h4>
<p>but Cargo handles a lot more I think than buck probably does (aka build scripts and such)</p>



<a name="166127409"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127409" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127409">(May 20 2019 at 23:06)</a>:</h4>
<p>we ended up with a model of each action produces some number of outputs</p>



<a name="166127415"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127415" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127415">(May 20 2019 at 23:06)</a>:</h4>
<p>and you depend on an action + an output</p>



<a name="166127418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127418">(May 20 2019 at 23:06)</a>:</h4>
<p>(e.g. the edges in the dependency graph are annotated with an output they depend on)</p>



<a name="166127434"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166127434" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166127434">(May 20 2019 at 23:07)</a>:</h4>
<p>and that way when an output is ready you delete all those edges and then just continuously schedule things that have no dependencies that aren't finished</p>



<a name="166128627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166128627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166128627">(May 20 2019 at 23:27)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> It would help if build scripts had proper dependencies, then it would be a clean mapping. I'm assuming that Cargo has to maintain a pile of pseudo dependencies to try and sequence <a href="http://build.rs" target="_blank" title="http://build.rs">build.rs</a> builds and execution correctly.</p>



<a name="166128654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166128654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166128654">(May 20 2019 at 23:27)</a>:</h4>
<p>Nah it's more just sequencing compiles appropriately that was the issue</p>



<a name="166128702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166128702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166128702">(May 20 2019 at 23:28)</a>:</h4>
<p>the deps issue is actually quite easy within cargo, this was just making sure we got as much parallelism as we could without tampering with cargo's entire backend</p>



<a name="166128894"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/195180-t-compiler/wg-pipelining/topic/Pipelining%20for%20Buck/near/166128894" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jeremy Fitzhardinge <a href="https://rust-lang.github.io/zulip_archive/stream/195180-t-compiler/wg-pipelining/topic/Pipelining.20for.20Buck.html#166128894">(May 20 2019 at 23:31)</a>:</h4>
<p>Yeah, but the build depends on running the buildscript for its side-effects, which means that it can't overlap the crate's build. Nor do you really know when it needs rebuilding.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>